Kernel-level attacks or malicious programs, such as rootkits, that compromise the kernel of an operating system are one of the most important concerns in systems security at present. These attacks can run at the same privilege level as the kernel and can thus modify kernel-level code or sensitive data to hide various malicious activities, to change operating system behavior, or even to take complete control of the system. Kernel-level security tools can be crippled and made ineffective by these attacks, which can run, access, and modify these security tools. A large body of research has adopted virtual machine (VM) monitor technology in an effort to mitigate such attacks. A higher privileged hypervisor outside of a virtual machine in which the kernel runs can enforce memory protections and preemptively intercept events throughout the operating system environment.
A major reason for adopting virtualization is to isolate security tools from an untrusted VM by moving those security tools to a separate, trusted, secure VM, and then using introspection to monitor and protect the untrusted VM from inside the trusted VM. Approaches that passively monitor various security properties, by periodically looking inside the untrusted VM for evidence of suspicious activity, have been proposed, but passive monitoring can only detect remnants of an already successful attack. Active monitoring from outside of the untrusted VM, in contrast, has the advantage of detecting attacks earlier and preventing certain attacks from succeeding. Active monitoring from outside of an untrusted VM can be achieved by placing secure hooks inside the untrusted VM, to intercept various events and invoke the security tool residing in a separate secure VM. Because the secure VM is isolated from the untrusted VM, so as to prohibit tampering, switching between the VMs occurs through the hypervisor. But the large overhead for switching between the untrusted VM, the hypervisor, and the secure VM makes this approach suitable only for actively monitoring a few events that occur less frequently during system execution.
Thus, with previous systems and methods, in-VM monitoring provides an inadequate level of security, while monitoring from outside a VM is feasible only when limiting the number and type of events that can be actively monitored, and.